Intuitive and Lightweight User Interaction for Medical Augmented Reality

The graphical overlay of additional medical information over the patient during a surgical procedure has long been considered one of the most promising applications of augmented reality. While many experimental systems for augmented reality in medicine have reached an advanced state and can deliver high-quality augmented video streams, they usually depend heavily on specialized dedicated hardware. We have described a novel medical augmented reality application, which is based almost exclusively on existing, commercially available, and certified medical equipment [5]. In this paper, we show how the capabilities of a so-called image guided surgery system for tracking surgical instruments can be utilized for user interaction. Our method enables the user to define points and freely drawn shapes in 3D and provides selectable menu items, which can be located in immediate proximity to the patient. This eliminates the need for conventional touchscreenor mousebased user interaction without requiring additional dedicated hardware like dedicated tracking systems or specialized 3D input devices. We demonstrate our new input method with an application for creating operation plan sketches directly on the patient in an augmented view.